Wittenberg University - Master of Science in Analytics

ANLT 510 - Advanced Statistics and Modeling

Day 1: Getting Comfortable with Uncertainty

05 Sep 2021

Course Overview

Basics Course Info

Course Overview

Course Pre-requisites & Co-requisites

Course learning objectives

Course Format, Resources & Software

Course schedule (tentative)

Assessment

Late Assignments

Reproducible Solutions

Machine Learning Algorithms

Supervised Vs Unsupervised Learning Algorithms

Classes of Machine learning algorithms with applications (reference: https://medium.com/@sanchittanwar75/introduction-to-machine-learning-and-deep-learning-bd25b792e488)

Classes of Machine learning algorithms with applications (reference: https://medium.com/@sanchittanwar75/introduction-to-machine-learning-and-deep-learning-bd25b792e488)

So, when would an unsupervised learning algorithm be used?

Supervised learning algorithms can be divided into two main classes

Regression algorithms: when the output/response variable is numeric and continuous

Classification algorithms: when the output/response variable is discrete or categorical

Model Complexity vs. Model Accuracy

The big picture of supervised learning algorithms

Common elements of supervised learning algorithms

Supervised learning Example

Elements of supervised learning algorithms: #1 Data

Plot of some ideal data

Plot of some ideal data

Elements of supervised learning algorithms: #2 an assumed form of \(f_{\text{imperfect}}\)

Fitting the ideal data with a "perfect" model

Fitting the ideal data with a “perfect” model

Elements of supervised learning algorithms: #3 parameter values

Elements of supervised learning algorithms: #4 loss functions

Supervised Learning Example (Cont.)

Visualizing loss functions

A Naive loss function

\[ Loss_{_{naive}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N y_i-m\times x_i-b. \]

loss_naive <- function(params,x,y) {

  if(length(params) != 2) stop("Params should be a length 2 vector")
  
  m <- params[1]
  b <- params[2]
  
  return(sum(y - m * x - b))

}
optim(par = c(1,1),
      fn = loss_naive,
      x = df$x,
      y = df$y,
      control = list(fnscale = 1))
$par
[1] 2.684172e+55 4.996335e+54

$value
[1] -1.111779e+57

$counts
function gradient 
     501       NA 

$convergence
[1] 1

$message
NULL



A good loss function

\[ Loss_{_{absolute}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N \Big\vert y_i-m\times x_i-b\Big\vert. \]

# First define a function to optimize
loss_absolute <- function(params,x,y) {

   if(length(params) != 2) stop("Params should be a length 2 vector")
  
   m <- params[1]
   b <- params[2]
   
   return(sum(abs(y - m * x - b)))

}
optim(par = c(1,1),           # provide starting values for m and b
      fn = loss_absolute,     # define function to optimize
      x = df$x,               # provide values for known parameters
      y = df$y,               # provide values for known parameters
      control = list(fnscale = 1))
$par
[1] 5 3

$value
[1] 3.301596e-06

$counts
function gradient 
     121       NA 

$convergence
[1] 0

$message
NULL



A better loss function

\[ Loss_{_{convex}}(\mathbf{y},\mathbf{x},m,b) = \sum_{i=1}^N \Big( y_i-m\times x_i-b\Big)^2. \]

loss_convex <- function(params,x,y) {

  if(length(params) != 2) stop("Params should be a length 2 vector")
  
  m <- params[1]
  b <- params[2]
  
  return(sum((y - m * x - b) ^ 2))    

}
optim(par = c(1,1),        # provide starting values for m and b
      fn = loss_convex,    # define function to optimize
      x = df$x,            # provide values for known parameters
      y = df$y,            # provide values for known parameters
      control = list(fnscale = 1))
$par
[1] 4.999800 3.000373

$value
[1] 2.501828e-06

$counts
function gradient 
      71       NA 

$convergence
[1] 0

$message
NULL



How can I know which algorithm to choose?